Evaluation of Two Levels of Automation

Two versions of an electronic checklist have been implemented in the Advanced Concepts Flight Simulator (ACFS) at NASA Ames Research Center. The two designs differ in the degree of pilot involvement in conducting the checklists. One version (manual-sensed), requires the crew to manually acknowledge the completion of each checklist item. The other version (automatic-sensed), automatically indicates completed items without requiring pilot acknowledgement. These two designs and a paper checklist (as a control condition) were evaluated in line-oriented simulation. Twelve air crews from one major air carrier flew a routine, four-leg, short-haul trip. This paper presents and discusses the portion of the experiment that was concerned with measuring the effect of the degree of automation on the crews’ performance. It discusses and presents evidence for a potential down side of implementing an electronic checklist that is designed to provide fully redundant monitoring of human procedure execution and monitoring.